Good parallel software development practices. Apache Spark case
نویسنده
چکیده
Recently, Spark as data processing engine, gained huge popularity because of better performance in terms of the speed. Developers of Spark claim that it may outperform Hadoop MapReduce in 100 times in memory and 10 times on disk [1]. This paper outlines which innovations improved speed and how. In order to investigate improvements, I analysed technical documentation, which is available, since both projects are opensource, and tested performance by myself. Indeed, not only the ability of Spark to run in-memory influences the speed, but also Spark enhanced architecture, so set of MapReduce overheads were eliminated.
منابع مشابه
Large-scale virtual screening on public cloud resources with Apache Spark
BACKGROUND Structure-based virtual screening is an in-silico method to screen a target receptor against a virtual molecular library. Applying docking-based screening to large molecular libraries can be computationally expensive, however it constitutes a trivially parallelizable task. Most of the available parallel implementations are based on message passing interface, relying on low failure ra...
متن کاملScalability Potential of BWA DNA Mapping Algorithm on Apache Spark
This paper analyzes the scalability potential of embarrassingly parallel genomics applications using the Apache Spark big data framework and compares their performance with native implementations as well as with Apache Hadoop scalability. The paper uses the BWA DNA mapping algorithm as an example due to its good scalability characteristics and due to the large data files it uses as input. Resul...
متن کاملSystemML: Declarative Machine Learning on Spark
The rising need for custom machine learning (ML) algorithms and the growing data sizes that require the exploitation of distributed, data-parallel frameworks such as MapReduce or Spark, pose significant productivity challenges to data scientists. Apache SystemML addresses these challenges through declarative ML by (1) increasing the productivity of data scientists as they are able to express cu...
متن کاملMulti-objective Big Data Optimization with jMetal and Spark
Big Data Optimization is the term used to refer to optimization problems which have to manage very large amounts of data. In this paper, we focus on the parallelization of metaheuristics with the Apache Spark cluster computing system for solving multi-objective Big Data Optimization problems. Our purpose is to study the influence of accessing data stored in the Hadoop File System (HDFS) in each...
متن کاملMassively Parallel Algorithms and Hardness for Single-Linkage Clustering Under $\ell_p$-Distances
We present massively parallel (MPC) algorithms and hardness of approximation results for computing Single-Linkage Clustering of n input d-dimensional vectors under Hamming, `1, `2 and `∞ distances. All our algorithms run in O(logn) rounds of MPC for any fixed d and achieve (1 + )-approximation for all distances (except Hamming for which we show an exact algorithm). We also show constant-factor ...
متن کامل